Skip to content

test: add extension PipelinePolicy tests#991

Open
silvi-t wants to merge 18 commits into
Kuadrant:mainfrom
silvi-t:extension-PipelinePolicy-tests
Open

test: add extension PipelinePolicy tests#991
silvi-t wants to merge 18 commits into
Kuadrant:mainfrom
silvi-t:extension-PipelinePolicy-tests

Conversation

@silvi-t
Copy link
Copy Markdown
Contributor

@silvi-t silvi-t commented Jun 1, 2026

Important

This PR modifies shared/core testsuite code that could potentially affect multiple test areas. 2 reviewers should review this PR to ensure adequate coverage.

Description

  • Add comprehensive test coverage for the PipelinePolicy extension (57 tests across 14 test files)
  • Add PipelinePolicy client implementation for creating and managing PipelinePolicy CRDs
  • Add pipeline_policy_extension_service image configuration for the ThreatAssessmentService gRPC backend
  • Mark known failing tests with xfail linked to upstream issues for policy isolation, deletion reconciliation, top-level fail action, deny with CEL body, and auth+deny timeout

Changes

New: PipelinePolicy Client

  • testsuite/kuadrant/extensions/pipeline_policy.pyPipelinePolicy class with methods for deny, fail, response headers, gRPC method actions, and action method definitions

New: Test Infrastructure

  • conftest.py — shared fixtures: pipeline_policy, threat_assessment_service, CRD existence check (check_pipeline_policy_crd)
  • interactions/conftest.py — fixtures for AuthPolicy/RateLimitPolicy interaction tests: authorization, auth, rate_limit
  • config/settings.yaml / config/settings.local.yaml.tpl — added pipeline_policy_extension_service.image setting
  • testsuite/utils/constants.py — added THREAT_ASSESSMENT_THRESHOLD and EXTENSION_POLICY_PROPAGATION_WAIT

Known Issues (xfail) - xfails has been removed; these issues no longer exist

Verification steps

Run all PipelinePolicy tests sequentially:

poetry run pytest -vv -n4 testsuite/tests/singlecluster/extensions/pipeline_policy/

Closes #974, #975, #976

Summary by CodeRabbit

Release Notes

  • New Features

    • Added PipelinePolicy extension support enabling request and response pipeline actions including deny rules, fail conditions, header manipulation, and gRPC integration for threat assessment.
    • Added configuration support for threat assessment service integration.
  • Tests

    • Added comprehensive test suite covering PipelinePolicy functionality: basic operations, action composition, AuthPolicy and RateLimitPolicy interactions, lifecycle management, error handling, and isolation verification.

crstrn13 and others added 12 commits May 27, 2026 12:47
Signed-off-by: Alexander Cristurean <acristur@redhat.com>
Signed-off-by: Alexander Cristurean <acristur@redhat.com>
Signed-off-by: Alexander Cristurean <acristur@redhat.com>
Signed-off-by: Alexander Cristurean <acristur@redhat.com>
Signed-off-by: Alexander Cristurean <acristur@redhat.com>
Signed-off-by: Alexander Cristurean <acristur@redhat.com>
This reverts commit 468dddc.

Signed-off-by: Alexander Cristurean <acristur@redhat.com>
Signed-off-by: Alexander Cristurean <acristur@redhat.com>
Signed-off-by: Alexander Cristurean <acristur@redhat.com>
Signed-off-by: Alexander Cristurean <acristur@redhat.com>
Signed-off-by: Alexander Cristurean <acristur@redhat.com>
Signed-off-by: Silvia Tarabova <starabov@redhat.com>
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Jun 1, 2026

Review Change Stack

Warning

Review limit reached

@silvi-t, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 51 minutes and 40 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 2bf7707e-3ffd-4e1a-b49c-50da5e0c004e

📥 Commits

Reviewing files that changed from the base of the PR and between f60dcf7 and edb84c8.

📒 Files selected for processing (7)
  • testsuite/tests/singlecluster/extensions/pipeline_policy/conftest.py
  • testsuite/tests/singlecluster/extensions/pipeline_policy/interactions/test_pipeline_policy_auth.py
  • testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_composition.py
  • testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_deny.py
  • testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_isolation.py
  • testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_lifecycle.py
  • testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_validation.py
📝 Walkthrough

Walkthrough

Adds a PipelinePolicy Python model, config/constants, shared fixtures, and a comprehensive integration test suite exercising request/response actions, gRPC upstreams, targeting/isolation, lifecycle and validation, and interactions with Auth and RateLimit policies.

Changes

PipelinePolicy Extension Testing Infrastructure

Layer / File(s) Summary
Configuration and constants
Makefile, config/settings.local.yaml.tpl, config/settings.yaml, testsuite/utils/constants.py
Adds pipelinepolicies.extensions.kuadrant.io to clean targets, config entries for pipeline_policy_extension_service image, and constants THREAT_ASSESSMENT_THRESHOLD and EXTENSION_POLICY_PROPAGATION_WAIT.
PipelinePolicy model and base fixtures
testsuite/kuadrant/extensions/pipeline_policy.py, testsuite/tests/singlecluster/extensions/pipeline_policy/conftest.py
Introduces ActionSection and PipelinePolicy classes and shared fixtures for CRD checks, instance creation, threat-assessment service deployment, commit/wait and cleanup.
Auth & RateLimit fixtures and interaction tests
testsuite/tests/singlecluster/extensions/pipeline_policy/interactions/*
Adds OIDC auth and rate-limit fixtures plus tests exercising PipelinePolicy interactions with AuthPolicy and RateLimitPolicy, including combined scenarios.
Basic and composition tests
testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_basic.py, test_pipeline_policy_composition.py
Happy-path and composition tests validating action ordering, short-circuit semantics, gRPC variable usage, empty and one-sided pipelines.
gRPC upstream and fail tests
testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_grpc.py, test_pipeline_policy_grpc_errors.py, test_pipeline_policy_fail.py
Validates gRPC action-method wiring, conditional invocation, threat-assessment integration, error cases and fail short-circuit behaviour.
Deny, response actions, targeting & isolation tests
testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_deny.py, test_pipeline_policy_response.py, test_pipeline_policy_targeting.py, test_pipeline_policy_isolation.py
Covers request/response denies, response header injections, gateway/route targeting, and isolation to ensure policies affect only intended routes/gateways.
Lifecycle and validation tests
testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_lifecycle.py, test_pipeline_policy_validation.py
Validates update/delete propagation and multiple validation failure modes (invalid targets, CEL errors, gRPC variable issues).

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related issues

Possibly related PRs

Suggested labels

Test case

Suggested reviewers

  • emmaaroche
  • azgabur
  • crstrn13

"🐰 I hopped through code with zest and cheer,
New policies and tests now appear,
gRPC calls, denies, headers in line,
Fixtures ready, assertions fine,
Hop on — the pipeline tests are here!"

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The PR description lacks critical information required by the template structure. Missing formal sections for Description, Changes, and Verification following the specified format and PR Title Guidelines. Reformat the PR description to follow the template structure: add a formal Description section, organize Changes under clear subsections, include explicit Verification steps in the required format, and ensure the PR title strictly adheres to conventional commit format without additional commentary.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The PR title 'test: add extension PipelinePolicy tests' is fully related to the changeset, clearly summarizing the main objective of adding comprehensive test coverage for the PipelinePolicy extension.
Linked Issues check ✅ Passed The code changes fully align with linked issue #974 objectives: PipelinePolicy client implementation, pytest fixtures, configuration settings, test infrastructure, and 57 comprehensive tests across 14 test files covering all action types and interactions.
Out of Scope Changes check ✅ Passed All changes are within scope: configuration updates, PipelinePolicy client/fixtures, test infrastructure, and test cases—all directly supporting the linked issue objectives for PipelinePolicy extension testing.
Docstring Coverage ✅ Passed Docstring coverage is 98.02% which is sufficient. The required threshold is 80.00%.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@silvi-t silvi-t self-assigned this Jun 1, 2026
@silvi-t silvi-t added this to Kuadrant Jun 1, 2026
@silvi-t silvi-t moved this to In Progress in Kuadrant Jun 1, 2026
Signed-off-by: Alexander Cristurean <acristur@redhat.com>
@silvi-t silvi-t changed the title Extension pipeline policy tests test: add extension PipelinePolicy tests Jun 1, 2026
silvi-t added 3 commits June 1, 2026 16:41
…tests to subdirectory

Signed-off-by: Silvia Tarabova <starabov@redhat.com>
…e test isolation

Signed-off-by: Silvia Tarabova <starabov@redhat.com>
…top-level-fail validation tests

Signed-off-by: Silvia Tarabova <starabov@redhat.com>
@silvi-t silvi-t force-pushed the extension-PipelinePolicy-tests branch from 3556628 to ce39290 Compare June 2, 2026 08:41
Signed-off-by: Silvia Tarabova <starabov@redhat.com>
@silvi-t silvi-t marked this pull request as ready for review June 2, 2026 10:04
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (3)
testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_validation.py (1)

69-73: TODO: assert the expected validation message once available.

Until the message is asserted, this test only checks Accepted=False, which could pass for an unrelated rejection reason. Tighten it when the upstream validation lands.

Would you like me to open a tracking issue for adding the message assertion here?

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_validation.py`
around lines 69 - 73, Test currently only asserts
policy.wait_until(has_condition("Accepted", "False"), ...) which can match
unrelated rejections; update the test to also assert the expected validation
message once the upstream validation is implemented by replacing the TODO with
an assertion that the policy's condition contains the exact expected message
(use policy.refresh().model.status.conditions to read conditions) — for example,
extend has_condition usage or add an assert that any(c.type == "Accepted" and
c.status == "False" and expected_message in c.message for c in
policy.refresh().model.status.conditions) where expected_message is the upstream
validation text; leave a TODO/issue reference if the message is not yet
available.
testsuite/tests/singlecluster/extensions/pipeline_policy/interactions/test_pipeline_policy_ratelimit.py (1)

8-22: ⚖️ Poor tradeoff

Optional: consider centralising the duplicated pipeline_policy/commit fixtures.

The header-only pipeline_policy fixture and the commit autouse fixture are repeated (with small variations in committed components) across the three interaction modules. A shared parametrised helper in interactions/conftest.py could reduce drift, though the per-module differences make this a low priority.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@testsuite/tests/singlecluster/extensions/pipeline_policy/interactions/test_pipeline_policy_ratelimit.py`
around lines 8 - 22, Extract the duplicated pipeline_policy fixture and the
autouse commit fixture into a shared interactions/conftest.py: move the
header-only pipeline_policy setup (the pipeline_policy fixture that calls
pipeline_policy.on_http_response.add_headers([...]) and returns pipeline_policy)
and the commit fixture (which iterates components, calls
request.addfinalizer(component.delete), component.commit(), and
component.wait_for_ready()) into conftest.py, and make commit parametrisable so
callers can pass which components to commit (e.g., via pytest param or an
injected fixture) while preserving autouse behavior where needed; update the
three interaction test modules to import/rely on the shared pipeline_policy and
to call the commit fixture with the appropriate component list to avoid
duplication.
testsuite/kuadrant/extensions/pipeline_policy.py (1)

77-83: 💤 Low value

Inconsistent header serialisation between add_headers and add_deny.

add_headers serialises the headers list via str(headers), producing a Python repr with single quotes (e.g. [['x-single', 'one']]), whereas add_deny(with_headers=...) expects a caller-supplied double-quoted string (e.g. '[["x-deny-reason", "blocked"]]'). This only works if the extension parses headersToAdd/withHeaders as CEL (single quotes valid) rather than strict JSON. Consider accepting a List[List[str]] in both methods and serialising consistently to avoid surprises for future callers.

Please confirm the extension accepts a Python-style single-quoted list for headersToAdd (i.e. CEL evaluation rather than JSON parsing); the response-header tests should fail if it does not.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@testsuite/kuadrant/extensions/pipeline_policy.py` around lines 77 - 83, The
add_headers method currently serialises headers with Python's str(headers)
producing single quotes, causing inconsistency with add_deny's expected
double-quoted JSON; change add_headers (and ensure add_deny) to accept headers
as a List[List[str]] and serialise them with json.dumps before placing into the
action dict (keys "headersToAdd" and "withHeaders"), preserving predicate
handling and the `@modify` decorator so both methods produce consistent JSON-style
double-quoted strings for downstream parsing.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@testsuite/kuadrant/extensions/pipeline_policy.py`:
- Around line 77-83: The add_headers method currently serialises headers with
Python's str(headers) producing single quotes, causing inconsistency with
add_deny's expected double-quoted JSON; change add_headers (and ensure add_deny)
to accept headers as a List[List[str]] and serialise them with json.dumps before
placing into the action dict (keys "headersToAdd" and "withHeaders"), preserving
predicate handling and the `@modify` decorator so both methods produce consistent
JSON-style double-quoted strings for downstream parsing.

In
`@testsuite/tests/singlecluster/extensions/pipeline_policy/interactions/test_pipeline_policy_ratelimit.py`:
- Around line 8-22: Extract the duplicated pipeline_policy fixture and the
autouse commit fixture into a shared interactions/conftest.py: move the
header-only pipeline_policy setup (the pipeline_policy fixture that calls
pipeline_policy.on_http_response.add_headers([...]) and returns pipeline_policy)
and the commit fixture (which iterates components, calls
request.addfinalizer(component.delete), component.commit(), and
component.wait_for_ready()) into conftest.py, and make commit parametrisable so
callers can pass which components to commit (e.g., via pytest param or an
injected fixture) while preserving autouse behavior where needed; update the
three interaction test modules to import/rely on the shared pipeline_policy and
to call the commit fixture with the appropriate component list to avoid
duplication.

In
`@testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_validation.py`:
- Around line 69-73: Test currently only asserts
policy.wait_until(has_condition("Accepted", "False"), ...) which can match
unrelated rejections; update the test to also assert the expected validation
message once the upstream validation is implemented by replacing the TODO with
an assertion that the policy's condition contains the exact expected message
(use policy.refresh().model.status.conditions to read conditions) — for example,
extend has_condition usage or add an assert that any(c.type == "Accepted" and
c.status == "False" and expected_message in c.message for c in
policy.refresh().model.status.conditions) where expected_message is the upstream
validation text; leave a TODO/issue reference if the message is not yet
available.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: d003d1c4-2f1a-4a81-ac35-7eac9f6ba6e9

📥 Commits

Reviewing files that changed from the base of the PR and between 4796963 and 25a4747.

📒 Files selected for processing (23)
  • Makefile
  • config/settings.local.yaml.tpl
  • config/settings.yaml
  • testsuite/kuadrant/extensions/pipeline_policy.py
  • testsuite/tests/singlecluster/extensions/pipeline_policy/__init__.py
  • testsuite/tests/singlecluster/extensions/pipeline_policy/conftest.py
  • testsuite/tests/singlecluster/extensions/pipeline_policy/interactions/__init__.py
  • testsuite/tests/singlecluster/extensions/pipeline_policy/interactions/conftest.py
  • testsuite/tests/singlecluster/extensions/pipeline_policy/interactions/test_pipeline_policy_auth.py
  • testsuite/tests/singlecluster/extensions/pipeline_policy/interactions/test_pipeline_policy_auth_ratelimit.py
  • testsuite/tests/singlecluster/extensions/pipeline_policy/interactions/test_pipeline_policy_ratelimit.py
  • testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_basic.py
  • testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_composition.py
  • testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_deny.py
  • testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_fail.py
  • testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_grpc.py
  • testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_grpc_errors.py
  • testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_isolation.py
  • testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_lifecycle.py
  • testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_response.py
  • testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_targeting.py
  • testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_validation.py
  • testsuite/utils/constants.py

@silvi-t silvi-t requested a review from a team June 2, 2026 10:18
@silvi-t silvi-t moved this from In Progress to Ready For Review in Kuadrant Jun 2, 2026
Comment on lines +34 to +39
policy.add_action_method(
name="assess",
url=svc_url,
service="threat.v1.ThreatAssessmentService",
method="AssessRequest",
message_template="threat.v1.ThreatRequest{uri: request.path}",
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this bit is reused in a lot of tests below. Thinking maybe we could wrap this into some sort of function, maybe the whole pipelinepolicy creation, similar to how other fixtures, such as api_key are being handels.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried wrapping this into a fixture (similar to create_api_key), but was getting random 500 status codes from the server. If you have a good solution for this we can implement it, but I wasn't able to make the tests reliable with the fixture approach.

@silvi-t silvi-t force-pushed the extension-PipelinePolicy-tests branch from f60dcf7 to 89a0f0f Compare June 4, 2026 08:45
Signed-off-by: Silvia Tarabova <starabov@redhat.com>
@silvi-t silvi-t force-pushed the extension-PipelinePolicy-tests branch from 89a0f0f to edb84c8 Compare June 4, 2026 08:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Ready For Review

Development

Successfully merging this pull request may close these issues.

Extensions SDK testing: PipelinePolicy (out-of-tree extension)

2 participants